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Abstract 



This study examined three questions about measures of accomplishments— notable attainments 
that have been publicly recognized: their pseudoipsativity, the correspondence between 
quantity and quality scores, and their dimensionality. Comparable samples of graduate 
students described their accomplishments on a questionnaire or judged the similarity of the 
same accomplishments. Accomplishments in the same field were positively correlated, while 
accomplishments in different fields varied in their correlations, some being positive and others 
being negative; these results are inconsistent with the predominantly negative correlations that 
would occur with ipsative measures. Measures of the quantity and quality (importance, rarity) 
of accomplishments correlated highly, after correction for attenuation, and appeared to assess 
the same thing. And analyses of the self-report data found that accomplishments were 
factorially complex, with many of the same factors identified in other kinds of measures 
appearing; the factors observed in the judgment data largely corresponded to the semantic 
features of the accomplishment items. 
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Measuring Accomplishments: Pseudoipsativity, Quantity vs. Quality, and Dimensionality 
It is universally recognized that what a person has accomplished in life provides 
important information about him or her. On the one hand, the accomplishments are an 
indication of the person’s abilities, motives and interests, and personality traits. On the other 
hand, the accomplishments forecast what he or she can achieve in the future. 

The importance of accomplishments for the systematic study of individual difference is 
underscored by criticisms of traditional measures and by calls for the assessment of real-life 
behavior. Ability tests, interest and personality inventories, projective techniques, and other 
methods, by and large, are measures of "signs" rather than "samples" of behavior 
(Goodenough, 1949). Although these methods have value, they also have limitations, leading 
a number of critics to argue for the importance of studying behavior directly, not its signs 
(e.g., Frederiksen, 1984; McClelland, 1973; Wernimont & Campbell, 1968). 

Beginning with the ground breaking research by Holland (1961), accomplishments-- 
commonly defined as notable attainments that have been publicly recognized— have been 
widely studied in high school, college, and graduate school populations, primarily as a means 
of assessing talent. This research has established that (a) accomplishments at early stages of 
the school career (e.g., high school) predict accomplishments at later stages (e.g., college); (b) 
early accomplishments are better predictors of later accomplishments than are tests of 
academic ability, school grades, interest inventories, or personality measures (personality 
scales and self-ratings); and (c) tests of academic ability are uncorrelated with 
accomplishments, but interest inventories and personality measures are modestly correlated 
with them (see the reviews by American College Testing Program, 1973; Baird, 1976). 

This body of work clearly suggests that the assessment of accomplishments has 
promise. Nonetheless, some important measurement issues remain to be resolved. One 



concerns the unusual psychometric properties of accomplishments measures. 

Accomplishment measures may be "pseudoipsative" in nature (Werts, 1967). The 
measurement process does not explicitly involve intra-individual comparisons that produce 
literal ipsativity (Cattell, 1944; Hicks, 1970; Radcliffe, 1963), but an analogous 
pseudoipsativity may occur because of the finite nature of people’s time and energy. A 
person who devotes himself or herself to one activity (e.g., writing novels) may simply not 
have the resources to do another activity (e.g., writing poems, doing scientific experiments). 
The upshot is that associations of accomplishment measures with each other, even those 
within the same area, as well as with other variables may be affected, paralleling what 
happens with explicit ipsative measures (e.g., predominantly negative intercorrelations among 
ipsative measures and near-zero average correlations with normative variables; Hicks, 1970; 
Radcliffe, 1963). Such pseudoipsativity could complicate studies of the reliability, 
dimensionality, and validity of accomplishment measures, as well as the procedures used to 
develop such devices. 

Another issue is the distinction between the quantity and quality of accomplishments. 
Although all accomplishments are uncommon and important, some are of higher quality than 
others. Whether the sheer quantity of accomplishments or their quality should be assessed is 
uncertain. Several studies have examined this issue. Most pertinent is an investigation by 
Skager, Schultz, and Klein (1965), who reported that a quantity measure, the number of all 
kinds of accomplishments, common or rare, correlated moderately with a quality measure (the 
quality of the most outstanding accomplishment) and had different patterns of correlations 
with ability tests and background variables. Two other studies are also relevant. Holland and 
Nichols (1964) found that measures of rare and common accomplishments correlated 



-3- 



moderately. And Nichols and Holland (1963) observed that a measure of rare 
accomplishments correlated substantially with a measure of all kinds of accomplishments (rare 
or common) and the two measures had different patterns of correlations with ability tests, 
personality measures, and background variables. The differential functioning of quantity and 
quality measures of accomplishments, and of rare and common accomplishments measures in 
these studies may be attributable to their unreliability. This possibility is in line with the 
Nichols and Holland (1963) conclusion, though Skager et al. (1965) argue otherwise. 

A third issue is the dimensionality of the accomplishments domain. Although a 
variety of accomplishments have been examined, no systematic effort has been made to map 
the domain. Research with interest inventories and biographical questionnaires suggests that 
the accomplishments domain may be equally complex (e.g., see the reviews by Dawes, 1991; 
Mumford & Owens, 1987). The unusual psychometric properties of accomplishments 
measures could complicate conventional methods of studying dimensionality, such as factor 
analyses of self-reports of accomplishments. An alternative approach to examining their 
dimensionality would be multivariate analyses of judgments of the similarity of 
accomplishments. 

Accordingly, the aims of this study were threefold: (a) to examine the 
pseudoipsativity of accomplishments measures; (b) to investigate the correspondence between 
quantity and quality scores for the measures; and (c) to explore the dimensionality of the 




measures. 



Method 



Overview 

Two comparable samples of graduate students were employed. One sample described 
their accomplishments on a questionnaire covering six broad areas of accomplishments; the 
other sample judged the similarity of these same accomplishments and rated their importance. 
The ipsativity of the accomplishments was assessed from their intercorrelations in the self- 
report data. The correspondence between quantity and quality scores was appraised from the 
correlations between quantity scores (the number of reported accomplishments) and two 
quality scores (the importance and rareness of reported accomplishments) in the self-report 
data. And the dimensionality of the accomplishments was evaluated by two alternative 
approaches, given the unusual psychometric properties of these variables: a factor analysis of 
a matrix of intercorrelations of the accomplishments in the self-report data and a parallel 
factor analysis of a matrix of similarity indexes in the similarity judgment data. 

Sample 

Two samples of graduate students (paid volunteers) were employed. Sample 1 
consisted of 204 first-year students, native English speakers from schools near 12 ETS 
computer-based test centers. This sample was obtained in a previous study (Bennett & Rock, 
1995). Sample 2 consisted of 75 students, U.S. citizens, from four institutions. The students 
in this sample were recruited to be comparable in sex, native language, and graduate field to 
those in Sample 1. Both samples were limited to students with usable data. The 
characteristics of the samples are summarized in Table 1. About a third of each sample was 
male (32%, 36%), all or virtually all were U.S. citizens (95%, 100%), and the most common 
graduate field was Social Science (43%, 49%). 
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Insert Table 1 about here 



Variables 

Accomplishments Questionnaire . The Accomplishments Questionnaire consisted of 52 
items covering six areas: academic achievement (5), leadership (5), practical language (public 
speaking, journalism; 12), aesthetic expression (creative writing, art, music, dramatics; 20), 
science (5), and mechanical (5). 1 These 6 are among 13 potential areas of accomplishments 
identified in a review of factor analyses of biographical, interest, and leisure-time activities 
data (Strieker, 1983). (The other areas are clerical, business, professional orientation, social 
welfare [e.g., coaching], religious, athletic, and adventure [e.g., sky diving]). The items used 
a combined multiple-choice and open-ended format (e.g., "Have you been elected to a major 
class office in college? If so, what position?"). This item format is modeled after one used 
previously by Skager et al. (1965) and Baird (1979). 

Item sorting . The multiple-choice portions of the 52 items on the Accomplishments 
Questionnaire were printed on 3" x 5" cards and arranged in random order. Subjects were 
given these written instructions for making judgements of their similarity, adapted from 
Strieker, Jacobs, and Kogan (1974): 

Your task is to judge the similarity of various activities and accomplishments. 

Each activity or accomplishment will be on a card. Please go through them and sort 
into groups the activities and accomplishments that seem similar. For example, 
"Organized a drive to lobby for the passage of legislation" and "Testified at a 
legislative hearing" might belong together in one group. And "Received an award 
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from a farm organization (e.g., Future Farmers of America)" and "Won a prize at a 
livestock show" might belong in another group. You may have as few or as many 
groups as you wish, and you may have as few or as many activities and 
accomplishments in a group as you wish. If an activity or accomplishment does not 
seem to belong in any of the groups, you may put it in the "miscellaneous" group. 

When you finish, put a rubber band around the cards in each group. And if you 
have a miscellaneous group, write "miscellaneous" on each of the cards in that group. 
Rating importance . The 52 multiple-choice portions of the items, arranged in the same 
order as in the sorting task, were administered in a questionnaire, with the following 
instructions: 

Please rate the merit of the following activities or accomplishments, using the 
10-point scale shown. [The ratings ranged from 1 (No Merit) to 10 (Greatest Merit).] 
Base the merit rating on the value and excellence of the activity or accomplishment. 
Procedures 

The Accomplishments Questionnaire was individually administered to the students in 
Sample 1 as part of a battery of other paper-and-pencil tests, as well as computer- 
administered tests. The sorting and rating tasks were individually administered, in 
counterbalanced order, to the students in Sample 2. 

Analysis 

Product-moment correlations were computed between the multiple-choice portions of 
the items on the Accomplishments Questionnaire (scored for endorsement, Yes = 1, No = 0). 

Quantity and quality scores were obtained for the Accomplishment Questionnaire’s 
aesthetic expression items, the area with the largest number of items (20). The quantity score 




-7- 



was the number of endorsed items. The two kinds of quality scores were (a) Very Important- 
-the number of endorsed items with high mean importance ratings (above the median rating of 
6.8) and Less Important--the number of endorsed items with low mean importance ratings (at 
or below the median of 6.8); and (b) Very Rare—the number of endorsed items with low 
endorsement proportions (at or below the median of .04) and Less Rare--the number of 
endorsed items with high endorsement proportions (above the median of .04). Product- 
moment correlations between the scores, as well as the scores’ internal-consistency reliability 
(Coefficient Alpha), were computed. 

The proportions of instances in which items were put in the same piles in the sorting 
task (Rosenberg & Jones, 1972) was tabulated. The square root of these proportions were 
calculated to obtain vector products (Nunnally, 1978). 

Parallel factor analyses were conducted of the matrix of intercorrelations from the 
questionnaire data and the matrix of vector products from the sorting data. This "proportion 
square root analysis" of the sorting data is a form of multidimensional scaling (Nunnally, 
1978). The principal components method was used in these analyses, alternative number of 
factors, suggested by the score test (Cattell, 1966), were rotated by the Varimax procedure 
(Kaiser, 1958), and the final decision about the number of factors was based on 
interpretability. Factors were matched by the coefficient of congruence (Harman, 1967) and 
visual inspection. 

Results and Discussion 

Item Intercorrelations 




The proportions of positive item intercorrelations within and between the six areas are 
shown in Table 2. The intercorrelations within areas were highly positive, the proportion of 
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positive correlations ranging from .64 for Aesthetic Expression to 1.00 for Academic 
Achievement and Science. The intercorrelations between areas were mixed, the proportion of 
positive correlations ranging from .23 (Aesthetic Expression and Mechanical) to .72 
(Leadership and Practical Language), with 8 of the 15 proportions being above .50. 

In short, most of the accomplishments were rarely reported. Accomplishments in the 
same field were positively related, and accomplishments in different fields varied in their 
relationships, some being positively related and others negatively related; there results within 
and between areas are inconsistent with the predominantly negative relations that would occur 
with ipsative measures (Hicks, 1970; Radcliffe, 1963). 



Insert Table 2 about here 



Intercorrelations of Quantity and Quality Scores 

The intercorrelations of the quantity and quality scores for the Aesthetic Expression 
area reported in Table 3. The internal-consistency reliability of these scores and the 
intercorrelations corrected for attenuation also appear in this table. 

The two counterparts of each kind of quality score correlated moderately: .46 for 
Very Important and Less Important, and .34 for Very Rare and Less Rare. In contrast, 
corresponding quality scores correlated substantially: .71 for Very Important and Very Rare, 
and .90 for Less Important and Less Rare. And all of the quality scores correlated very 
highly with the quantity score, the correlations ranging from .90 (Less Rare) to .72 (Very 
Rare). All of these correlations are attenuated by the unreliability of the scores. Corrected 
for attenuation, all of the correlations were above .90, except for the .60 for Very Rare and 
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Less Rare. It should be noted that the correlations between the quantity and quality scores 
and between the counterparts of each kind of quality score are inflated by item overlap. 

In brief, accomplishments differing in their quality were substantially related to each 
other and to the sheer quantity of accomplishments, when the unreliability of these measures 
was taken into account. 



Factor Analyses of Items 

The rotated factor loadings for the questionnaire data and the sorting data appear in 



Seven factors were identified in the questionnaire data. The factors accounted for 
19.7% of the total variance, ranging from 3.4% for Factor I to 2.4% for Factor VII. The 
factors were Art and Music (I), Radio and Television (II), Practical Language (III), 
Mechanical (IV), Public Speaking/Dramatic Arts (V), Science (VI), and Academic 
Achievement (VII). 

Four of the six original categorizations of the items based on previous factor analyses 
of biographical, interest, and leisure-time activities data (Strieker, 1983) were represented by 
factors in this analysis: Academic Achievement, Practical Language, Science, and 
Mechanical. A fifth categorization, aesthetic expression, was broken up into several factors: 
Art and Music, Public Speaking/Dramatic Arts, and Radio/Television. The only 
categorization not represented in any way was leadership. 



Insert Table 3 about here 



Table 4. 2 
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Eight factors were identified in the sorting data. The factors accounted for 44.9% of 
the total variance, ranging from 11.3% for Factor I to 1.8% for Factor VIII. The factors were 
Public Performance (I), College Activities (II), Paid Activities (III), Writing (IV), Prize 
Winning (V), Speaking (VI), Science (VII), and Artwork (VIII). 

Only one of the original categorizations of the items was represented by a factor: 
Science. Two other categorizations, academic achievement and leadership, were subsumed by 
a single factor, College Activities. Other categorizations were incompletely represented by 
factors: practical language by the Speaking factor, and aesthetic expression by the Artwork 
factor. 

It is noteworthy that most of the factors in this analysis corresponded to the semantic 
features of the items. For Public Performance (I) items, phrases such as "public 
performance," "publicly performed," "publicly broadcast," and "publicly shown"; for College 
Activities (II) items, "college"; for Paid Activities (III) items, "for pay, for a company or 
other organization"; for Prize Winning (V) items, "a winner or runner-up of a prize or 
award"; for Science (VII) items, "scientific"; and for Artwork (VIII) items, "artwork." The 
Writing (IV) and Speaking (VI) factors were important exceptions. Most of the items on the 
Writing factor included "wrote," but this word also appeared in items on other factors. All of 
the items on the Speaking factor concerned speaking but differed in their semantic content, 
their key phrases being "a formal speech," "master or mistress of ceremony," and 
"spokesperson or press aide." 

One factor in the two analyses matched: Science (VI in the questionnaire data, and 
VII in the sorting data). 
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In sum, a number of factors emerged in the analyses of the questionnaire and sorting 
data, and the factors generally differed in the two analyses, with most of those in the 
questionnaire data resembling factors previously identified in other kinds of measures and 
most of the factors in the sorting data corresponding to the semantic features of the items. 



Insert Table 4 about here 



Conclusions 

Pseudoipsativity 

No sign of pseudoipsativity in the accomplishments measures appeared, contrary to 
what was suggested previously (Werts, 1967). Substantively, the students in this study did 
not focus on one activity at the expense of competing activities. Whether this result is 
generalizable to other populations of examinees is uncertain. It is conceivable that other 
kinds of examinees may have a more single-minded devotion to a particular field, producing 
ipsativity in the accomplishments measures, but it should be recalled that the students in this 
study were graduate students, presumably more specialized in their interests than most 
examinees. 

These outcomes suggest that the unusual properties of accomplishments items do not 
interfere with the use and interpretation of standard statistical and psychometric analytical 
methods in their construction and evaluation. 

Quantity and Quality Scores 

The limited findings on the quantity vs. quality issue suggest that the two kinds of 
scores measure the same thing. Especially persuasive was the extremely high correspondence 
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between measures of the most important and least important accomplishments, after taking the 
measures’ unreliability into account. These results, supporting the Nichols and Holland 
(1963) contention that high and low level accomplishments function similarly, should not be 
overinterpreted. First, they were based on accomplishments in only one area, aesthetic 
expression. Whether the same results would occur in other areas is unknown. Second, the 
outcomes in analyses of this kind may depend on the quality of the pool of accomplishments 
examined: the greater the variability in the quality of accomplishments, the greater the 
opportunity for detecting differences among them. The accomplishments in the present 
analysis were generally at a high level. 

Dimensionality 

The factor analyses of the questionnaire data and the sorting data provide markedly 
different pictures of the dimensions underlying the accomplishments. Given the absence of 
pseudoipsativity, which would complicate analyses of the questionnaire data (Hicks, 1970; 
Radcliffe, 1963), and the correspondence between the factors in these data and in factor 
analyses of other kinds of measures, the results based on the questionnaire data appear to 
provide a better insight into what this device is actually measuring. It is evident that the 
accomplishments domain is factorially complex. It is also highly likely, in light of the 
general agreement between the factors identified in the questionnaire and in other kinds of 
measures, that additional accomplishments factors, resembling those found in other measures 
but not represented in the present questionnaire, exist and can be measured (clerical, business, 
professional orientation, social welfare, religious, athletic, and adventure; Strieker, 1983). 

The nature of the factors in the sorting data is intriguing. Despite the general 
correspondence observed between the factors and the semantic features of the items, it is far 
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from certain that the students based their judgments solely on the semantic content of the 
items, without regard to their substantive meaning. All of the factors with semantic content 
also had congruent substantive content, the semantic and substantive correspondence 
concerning the setting (college, for pay) or the activity (public performance, writing, prize 
winning, science, art). The possibility that the similarity judgments were based on substantive 
meaning, not semantic content, is also supported by the emergence of the Speaking factor, 
which is devoid of consistent semantic content, and the Writing factor, which shares the same 
semantic content with other factors. It seems implausible that subjects would base some of 
their judgments on semantics and others on substance. 
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Footnotes 

*A copy of the questionnaire is available from the first author. 

^he matrix of intercorrelations from the questionnaire data and the matrix of vector 
products from the sorting data are available from the first author. 
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Table 1 

Percentage Distributions of Background Characteristics, Samples 1 and 2 



Sample 



Variable 1 2 

(N=192) (N=75) 



Sex 

Male 

Female 

Graduate Major 
Humanities/Arts 
Education 
Social Sciences 
Life Sciences 
Physical Science 
Engineering 
Other 
Age 
20-29 
30-39 
40-49 



32 36 

68 64 



18 

14 
43 

15 
7 
1 
2 



12 

20 

49 

16 

3 

0 

0 



65 

27 

8 
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Table 1 (continued) 





Sample 




Variable 


1 


2 




(N=192) 


(N=75) 


Citizenship 






U.S. 


95 


100 


Other 


5 


0 



Note. The data in column 2 are from Bennett and Rock (1995). The sample is smaller than 
the one used in the present study (192 vs. 204) because students with missing data on other 
variables were excluded by Bennett and Rock. 
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Proportion of Positive Intercorrelations of Items in Six Areas. 
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Table 3 

Intercorrelations and Reliability of Quantity and Quality Scores for Aesthetic Expression Items 



Score 


(1) 


(2) 


( 3 ) 


( 4 ) 


( 5 ) 


(1) Standard (Quantity) 


(.66) 


.82 


.89 


.72 


.90 


(2) Very Important 


1.00 


(.49) 


.46 


.71 


.66 


(3) Less Important 


1.00 


.90 


(.52) 


.55 


.86 


(4) Very Rare 


1.00 


1.00 


.93 


(.66) 


.34 


(5) Less Rare 


1.00 


1.00 


1.00 


.60 


(.53) 



Note : Actual correlations appear above the diagonal; attentuation- corrected correlations appear 
below it. Diagonal values are internal-consistency reliability coefficients. 
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